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(54) Processing of images for autostereoscopic display 



(57) A method and display are disclosed in which a 
representation of a 3D model is provided for presenta- 
tion as a 3D image. The image may be presented under 
an array of spherical or lenticular microlenses so that 
different images are presented at different viewing an- 



gles. The images are rendered using a set of ortho- 
graphic projections; this can avoid the need for multiple 
cameras or the highly computer intensive processing 
associated with generation of simulated camera images 
and can also give improved results as compared to prior 
art multiview images. 
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Description 

[0001] This invention relates to the field of 3D dis- 
plays. 

[0002] Autostereoscopic 3D displays are a well- 
known class of 3D display that can be viewed without 
needing to wear special glasses. The general principle 
is to ensure that the image seen on a display screen is 
dependant on the position of the viewer. At least two im- 
ages are usually used, showing the desired view of the 
scene from two positions, corresponding to the positions 
of the left and right eye of the viewer for a limited range 
of head positions. The images are usually created by 
photographing a scene from positions corresponding to 
the centres of the intended eye positions. More than two 
images can be used, which allows a greater degree of 
head movement, whilst showing changes of parallax in 
the image, leading to increased realism. The images 
used for such a so-called multiview display are usually 
acquired by several cameras, each positioned so as to 
capture a view of the scene appropriate for the eye po- 
sition at which the image is visible. 
[0003] As an alternative to using several cameras, im- 
ages may be generated using computer graphics tech- 
niques to render images equivalent to those that would 
be captured by atwo or more real cameras. The process 
of rendering an image to simulate an image captured by 
a real camera is known as perspective projection. Im- 
ages rendered in this way must of course exhibit the 
usual properties of images obtained from a conventional 
camera, specifically that objects appear smaller when 
they are a long way away, and the angle at which the 
scene is viewed varies slightly across the image, by an 
amount equal to the angular width and height of the im- 
age. 

[0004] There are a number of known methods for re- 
alising an autostereoscopic display. For example, a 
sheet of lenticular lenses can be placed over a high-res- 
olution display. The image underneath each lenticular 
lens is formed by taking one column of pixels from each 
image in turn, so that the display presents the viewer 
with each image in sequence as the viewer's head 
moves parallel to the image. This process is illustrated 
in Fig. 2. An example of this "multiview" approach is de- 
scribed in "Design and applications of multiview 3D- 
LCD" by C. van Berkel et al, 1996 EuroDisplay Confer- 
ence, Birmingham, October 1996, and "Characterisa- 
tion and optimisation of 3D-LCD module design" by C. 
van Berkel and J. Clarke, SPIE International Confer- 
ence on Electronic Imaging, San Jose. 11-14 February 
1 997. A set of conventional cameras is used to capture 
the images. 

[0005] However, autostereoscopic displays based on 
these principles do not produce a fully accurate 3D rep- 
resentation of the scene; they merely presents the view- 
er with a number of discrete views. 
[0006] One problem is that often, the viewer will be 
sufficiently close to the screen that one of his eyes is 



presented with parts of one image from microlenses on 
the left of the screen and parts of another image from 
microlenses on the right, due to the changing angle be- 
tween the screen and the direction of view. These com- 

5 bined views will not represent an accurate portrayal of 
the scene. This is illustrated in Figure 4. It is possible to 
minimise this effect by careful design of the display sys- 
tem to account for the variation of angle for a given view- 
ing distance. An example of such a design is described 

10 in UK patent Application 2,272,597A (Sharp), which de- 
scribes an arrangement of illumination elements and cy- 
lindrical lenses which ensure that a given observer sees 
a single view across the whole of the display. However, 
this will only work for observers at a certain range of 

15 distances. 

[0007] There is a known method of generating an 
autostereoscopic display which makes no assumptions 
about the position of the viewer, and is capable of repro- 
ducing a 3D scene with high accuracy. This method is 

20 known as Integral Imaging, an example of which is de- 
scribed by McCormick et al., "Examination of the re- 
quirements for autostereoscopic, full parallax, 3D TV", 
IBC '94, IEE Conference Publication 397, September 
1 994, pp.477-482, and WO-A-953401 8. In this method, 

25 a special kind of camera captures an image which is 're- 
played' by placing a sheet of microlenses over the im- 
age. The camera consists of an arrangement of lenses, 
which can be thought of as behaving like an array of 
very small cameras, one positioned at the location of 

30 each microlens. When the image is replayed, each mi- 
crolens appears to have a colour and brightness that is 
a function of the angle at which it is viewed. The form of 
the image underneath the microlens array is such that 
the overall appearance is of a 3D image. The microlens 

35 array should ideally be composed of spherical micro- 
lenses, although lenticular lenses can also be used, in 
which case the image only exhibits parallax in the direc- 
tion normal to the lenticular elements. 
[0008] It is known that the image underneath the mi- 

40 crolens array can be generated by a computer, rather 
than by a special kind of camera. This allows a 3D image 
of a computer model to be generated. However, tech- 
niques for doing this have involved ray tracing and sim- 
ulation of the behaviour of the integral image camera, 

45 and are therefore complex and processor-intensive. For 
example, the image underneath each microlens can be 
computed by rendering an image of the scene from the 
viewpoint of that microlens, with an appropriate modifi- 
cation to allow objects that are on both sides of the im- 

50 age plane to be visible. For a typical microlens array 
consisting of, say 400 by 300 microlenses, this would 
require 120,000 separate rendering processes. 
[0009] The present invention seeks to provide - in one 
of its forms - a method of allowing conventional image 

55 rendering hardware to be used efficiently to produce an 
image to place underneath the microlens array, whilst 
enabling an accurate 3D representation of the scene to 
be provided. 
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Summary of The Invention 

[0010] Accordingly, the present invention consists, in 
one aspect, in a method of providing a viewable repre- 
sentation of a 3D model, comprising the steps of iden- 
tifying a set of different viewing angles and rendering 
from said model a set of orthographic projections corre- 
sponding respectively with said viewing angles. 
[0011] It is important to notethatthe present invention 
uses orthographic projections. An image rendered using 
orthographic projection is unlike an image captured with 
a conventional camera; indeed it is impossible to cap- 
ture such an image with a conventional camera. One 
fundamental difference is that the size of an object in an 
orthographic image projection is independent of its dis- 
tance from the camera; in conventional perspective pro- 
jection (or in a normal camera image), more distant ob- 
jects appear smaller (the so-called perspective effect). 
This difference comes about because the angle at which 
the scene is viewed in an orthographic image is constant 
across the whole image. The field-of-view is specified 
in terms of the width and height of the rendered image 
(measured in units of distance, e.g. feet/metres), rather 
than by a horizontal and vertical field of view (measured 
in angular units e.g. degrees). 
[0012] We have found that using orthographic images 
overcomes a major problem found with conventional 
multiview displays. Specifically, in such displays, as ex- 
plained earlier, the property that determines which im- 
age is visible from a particular viewing position at any 
point on the screen is the angle between the screen and 
the direction of view of that point, as shown in Fig. 4. 
This will tend to result in several views being seen 
across the width of the display, which is an undesirable 
property if each view corresponds to a normal perspec- 
tive camera image. As already mentioned, some dis- 
plays have proposed modifications to the basic arrange- 
ments in an attempt to take account of the changing 
viewing angle across the display, for example the use 
of varying lens offsets across the display, as described 
for example in GB 2,272, 597A. 

[0013] In direct contrast to such complex prior meth- 
ods of compensating for problems with conventional 
perspective images, the present invention uses ortho- 
graphic images. This might at first seem unsatisfactory 
as the viewer should be presented with a "natural" per- 
spective image. However, it has been appreciated ac- 
cording to the invention that the nature of a display will 
take account of the changing viewing angle of the viewer 
with respect to the display. Specifically, the display will 
allow portions of d/'/ferenforthographically-rendered im- 
ages to be visible across the display from a given view- 
ing point. Since the viewing angle is constant across 
each orthographic image, but differs from one ortho- 
graphic image to another, the changing viewing angle 
automatically provides the correct parameter to control 
the selection of which image is seen. It has been appre- 
ciated that this is precisely what is needed to create a 



natural (perspective) imagefrom the viewpoint of an ob- 
server, tailored to the particular location of the observer. 
Thus, although the individual images rendered are or- 
thographic, what the viewer will actually see will appear 

5 to have the correct perspective. 

[0014] A further advantage is that the orthographic 
projections may be rendered efficiently. Firstly, ortho- 
graphic projections may be simpler to perform than per- 
spective projections, as the distance from the source 

10 does not need to be taken into account. In a preferred 
arrangement, a greater benefit is that the number of ren- 
dering processes may be equal to the number of view- 
points, typically equal to the number of pixels under 
each micro lens, and this will normally be smaller than 

15 the total number of lenses. For example, in a lenticular 
display with a resolution of 1024x768 and 64 horizontal 
viewing angles, nearly a million rendering operations 
would be required with conventional processing but only 
64 rendering operations may be required with an em- 

20 bodiment, a saving of better than 1 0,000 fold. It will be 
appreciated, as described below, that each rendering 
operation may comprise multiple rendering processes, 
to take into account depth of field. 
[0015] A yet further advantage that may be attained 

25 in preferred embodiments is that orthographic projection 
may allow depiction of objects both behind and in front 
of the display plane whereas conventional perspective 
rendering normally can only display objects on one side 
of the display. This advantageous feature may be pro- 
se vided independently. 

[0016] Advantageously, the set of orthographic pro- 
jections are spatially multiplexed to provide a set of mi- 
cro images, each micro image containing one pixel from 
each of the projections. 

35 [0017] In another aspect, the present invention con- 
sists in representation of a3D scene or model for display 
utilising 2D display means, comprising a set of 2D im- 
ages representing respective orthographic projections 
of the 3D scene or model at respective different angles. 

40 [0018] In still another aspect, the present invention 
consists in a method of processing a viewable represen- 
tation of a 3D model or scene, comprising the steps of 
demultiplexing the representation to form a set of ortho- 
graphic projections corresponding respectively with dif- 

45 ferent viewing angles, and processing each projection. 
[0019] The invention extends to software and hard- 
ware for implementing the aspects and preferred fea- 
tures. Further aspects and preferred features are set out 
in the claims and/or in the following specific description. 

so [0020] Advantageously, the processed projections 
are remultiplexed. 

[0021] The present invention recognises that each 
pixel underneath a spherical microlens in a true integral 
image represents the brightness and colour of the scene 
55 when viewed from the position of that microlens, at an 
angle that is a function of the position of the pixel with 
respect to the focal point of the microlens. We will as- 
sume for now that the arrangement of pixels underneath 
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each microlens are the same for all microlenses. There- 
fore, for example, the top left-hand pixel underneath 
each microlens corresponds to the light ray leaving the 
scene at a given angle, and this angle is the same for 
all top left-hand pixels under all microlenses. The posi- 
tion at which the light ray intersects the display surface 
varies according to the position of the microlens, but the 
angle does notvary. Thus, if we were to takethetop left- 
hand pixel from under each microlens and assemble 
them into an image, we would obtain a kind of image of 
the scene, similar to that which would be obtained by a 
camera a very long way away, having a very narrow 
field-of-view, such that the angle of view was essentially 
unchanging across the image. 

[0022] An embodiment of the invention will now be de- 
scribed by way of example with reference to the accom- 
panying drawings in which: 

Figure 1 is a diagram illustrating the known principle 
of integral image display; 

Figure 2 is a diagram illustrating the known use of 
multiple cameras to generate a multi-view image 
viewed through a lenticular screen; 
Figure 3 is a diagram illustrating an embodiment of 
the present invention; 

Figure 4 illustrates the way in which different imag- 
es on a multi-view display are seen in regions of the 
display, depending on the angle between the ob- 
server and the screen in each region; and 
Figure 5 illustrates how multiple orthographic imag- 
es may be rendered for a range of angles close to 
the central angle for a particular view, so that the 
images may be added together to generate an im- 
age whose resolution reduces according to the dis- 
tance of objects from the display screen. 

[0023] Referring briefly to Figure 1 , this illustrates the 
known technique mentioned above, in which a special, 
integral image camera can provide a 2D representation, 
which when viewed through the matching microlens ar- 
ray, creates a 3D display with full parallax. 
[0024] Figure 2 illustrates the other known technique 
mentioned above in which multiple cameras are used to 
generate a multi-view image to be viewed through a len- 
ticular screen. 

[0025] A 3D display according to this invention may 
be realised using an arrangement shown in Fig. 3. The 
figure shows a cross-section view looking down onto the 
display; it is assumed that the lenses are spherical, and 
that the view from the side would look similar. The vol- 
ume imaged by each orthographic camera is represent- 
ed as a rectangle. The orientation of each viewing vol- 
ume corresponds to the angle of view of a particular pix- 
el under each microlens. For example, the orientation 
of the shaded rectangle, corresponding to 'camera' A, 
corresponds to the angle at which the pixels labelled 'A' 
are visible through the microlenses. The width of each 
view corresponds to the width of the display screen 



when viewed at that angle. The extent of each viewing 
volume to the front and behind of the display corre- 
sponds to the range of depths between which objects 
are to be visible. It will be noted that this can extend both 

5 in front of and behind the display, whereas this cannot 
easily be achieved with perspective rendering. Informa- 
tion describing the orientation and size of each viewing 
volume is given to the computer rendering system in or- 
der to allow it to compute each orthographic view. The 

10 image computed from the view corresponding to area A 
in Fig. 3 is shown as a row of pixels on the left-hand side 
of the viewing volume. These pixels are multiplexed with 
pixels from the other views and displayed on the appro- 
priate pixels underneath each microlens. 

15 [0026] Microlenses in the form of lenticular or convex 
lenses are described herein for convenience. However, 
the invention is not limited to such lenses and other 
means of spatially demultiplexing the display may be 
used. Forexample pinholes, diffraction gratings, fresnel 

20 lenses and the like may be used and the term micro- 
lenses as used herein is intended to encompass any ar- 
rangement having the optical properties of varying the 
pixels presented for view based on the angle at which 
the image is viewed (i.e. spatially demultiplexing the im- 

25 age). An advantage of pinholes, particularly having a 
size corresponding to the pixel spacing on the display, 
is that a smooth transition from one image to another 
may be obtained. This effect may also be achieved us- 
ing lenses positioned so that the display is behind the 

30 focal point by a small distance to generate a small 
amount of blurring between pixels. Alternatively, to gen- 
erate a smooth transition without requiring a large 
number of rendering operations, the image may be fil- 
tered and/or pixels may be interpolated, as described 

35 below. 

[0027] It has been appreciated that it is important to 
generate an image where the brightness/colour is a 
function of angle of view and this is conveniently 
achieved by using spatial demultiplexing as in a conven- 

40 tional multiview image. However this is not essential 
and temporal demultiplexing could be used. For exam- 
ple, a high frame rate display could be used in conjunc- 
tion with a device for projecting the display at differing 
angles, for example moving mirrors or lenses or moving 

45 light sources. In such a case, the different images ren- 
dered would be presented sequentially to the display as 
the effective viewing angle changed. Thus it will be ap- 
preciated that references to multiplexing and demulti- 
plexing should not be construed as limited to the spatial 

50 multiplexing and demultiplexing shown in the preferred 
embodiment. 

[0028] The arrows in Fig. 3 show how the pixel values 
from view A are used to form the first pixel underneath 
each microlens; the pixels from view B are similarly used 
55 to form the second pixel underneath each lens, and so 
on. For example, with a regular array of spherical mic- 
rolenses with a spacing of 8 pixels horizontally and 8 
pixels vertically, it would be necessary to render a total 
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of 64 orthographic images. The resolution of each or- 
thographic image would be equal to the number of mi- 
crolenses in the display. 

[0029] Orthographic projection, although not previ- 
ously considered for use in generation of multiview im- 
ages, has of course previously been considered math- 
ematically and have there exists a well-known method 
for generating an image of a 3D model held in a com- 
puterfor which the angle of view is constant for all pixels 
in the image. Orthographic projection is most commonly 
used in generating plans, for example of buildings from 
computer models or other cases where it is desirable for 
vertical and horizontal lines to appear as such and lines 
of the same length in theforeground and backgroundto 
appearthe same length rather than background objects 
appearingsmaller as they would in a perspective or"nat- 
ural" view of the object. Whilst this known use of ortho- 
graphic projection is of course for a quite different prob- 
lem, techniques used for such an application may be 
applied to generate images in embodiments of the in- 
vention. Orthographic projection is supported, for exam- 
ple, by the "OpenGL" 3D graphics programming inter- 
face, the "Open Inventor" programming API, and many 
3D graphics hardware systems. Therefore, embodi- 
ments of the present invention can provide through use 
of orthographic projection, a convenient and efficient 
method of generating the image to place behind a mic- 
rolens sheet in order to provide an accurate 3D view of 
a 3D scene. 

[0030] The operation of the orthographic camera may 
be represented as a 4x4 matrix, known as the projection 
matrix in openGL. This is described, for example, in the 
"OpenGL Reference Manual", published by Addison- 
Wesley. The standard form of the matrix used to create 
an orthographic view is: 



2/(r-l) 


0 


0 


Tx 


0 


2/(t-b) 


0 


Ty 


0 


0 


2/(f-n) 


Tz 


0 


0 


0 


1 



where 

t, b define the positions of the planes at the top and 
bottom of the viewing volume, being equal to the 
x values of the top and bottom planes respec- 
tively, and the planes being normal to the x (ver- 
tical) axis; 

l,r define the planes on the left and right of the view- 
ing volume, being equal to the y values of the 
left and right-hand planes respectively, and the 
planes being normal to the y axis; 

f,n define the planes on the far and near ends of the 
viewing volume, being equal to the z values of 
the far and near planes respectively, and the 
planes being normal to the z axis. The camera 
is positioned at the origin, looking along the z 



axis in the direction of -ve z. in accordance with 
the usual convention in OpenGL. 

5 Tx = - (r+l) / (r-l) 

Ty = - (t+b) / (t-b) 

10 

Tz = -(f+n)/(f-n) 

[0031] This matrix premultiplies the coordinates of 
points in the 3D model being viewed, these points being 
15 represented in homogeneous coordinates as a vector 
of the form [x, y, z,1] T . So a point (x ; y, z) is transformed 
to the position 

((2x-r-l)/(r-l),(2y-t-b)/(t-b)) 

in the rendered image, with x coordinates ranging from 
-1 on the left of the image to +1 on the right, and y rang- 
ing from -1 at the bottom to +1 at the top. Note that the 
coordinates of the point in the image are independent 
of z; this is the fundamental characteristic of an ortho- 
graphic projection. 

[0032] Although the best quality 3D images are likely 
to be obtained by using spherical microlenses, an alter- 
native arrangement is to use cylindrical microlenses, for 
example as used in the multiview LCD system shown in 
Fig. 2. This provides parallax only in the horizontal di- 
rection (if the lenticular lenses run vertically). However, 
lenticular lenses are cheaper and easierto manufacture 
than spherical microlens arrays, and may be preferable 
in some applications. 

[0033] The invention may also be used to generate 
images for such lenticular lens displays, by using a vir- 
tual camera which exhibits the properties of an ortho- 
graphic camera in the horizontal direction, and behaves 
like a perspective camera in the vertical direction. This 
may be achieved by using the expression for the y dis- 
play coordinate given by the conventional perspective 
camera transformation (see the OpenGL Reference 
manual), i.e. 

y = (-2ny/z + t+ b)/(b-t) 

Note that such a combined orthographic/perspective 
transformation cannot be implemented by multiplication 
with a viewing matrix, since the perspective transforma- 
tion requires that the w coordinate of the transformed 
point (the 4th component of the position, in homogene- 
ous coordinates) be equal to the depth value of the point 
(which is -z, since the camera looks in the direction of 
the negative z axis). OpenGL divides the transformed 
x, y, z coordinates by w to obtain screen coordinates; 
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this allows the size of objects to appear inversely pro- 
portional to their distance from the camera. In this ap- 
plication, we require this division to happen for they co- 
ordinate, but not the x coordinate. Therefore, the trans- 
formation we require in this case would have to be im- 
plemented by special-purpose hardware or software, 
since OpenGL divides all coordinates by w. 
[0034] Alternatively, when generating images for a 
lenticular display, it is possible to use a conventional or- 
thographic transformation to render the set of images, 
although it is appreciated that the image will not appear 
exactly right. Each orthographic image may be rendered 
with a vertical angle of view set to correspond to the 
most likely vertical angle of view, for example zero (view- 
ing the screen at right angles). The display will then ap- 
pear to show images which have the characteristics of 
orthographic images in the vertical direction (e.g. the 
height of objects will not change with their distance from 
the viewer) , whilst appearing correct in the horizontal di- 
rection. In some circumstances, this will be acceptable 
behaviour. It is noted that with a lenticular autostereo- 
scopic display, it is generally not possible to account cor- 
rectly for changes in the vertical position of the viewer 
(this applies to conventional lenticular displays), regard- 
less of how the images are generated, so at best a len- 
ticular display will only give fully correct images for view- 
points at a given height. Thus the imperfection in the 
method of the embodiment can be tolerated under sim- 
ilar circumstances to imperfections in existing lenticular 
display applications. 

[0035] In some applications, the arrangement of pix- 
els underneath the microlenses may vary across the dis- 
play, for example if the spacing between lenses is not 
an integer multiple of the pixel spacing. In this case, the 
orthographic cameras may be used to generate images 
on a regular pixel lattice under each lens, and the result- 
ing image data, after multiplexing, may be resampled 
using conventional filtering and sampling techniques. If 
the microlenses themselves are arranged in some way 
otherthan on a rectangular lattice, such as a hexagonal 
pattern, then the orthographically-rendered images may 
be resampled to provide pixel data at locations corre- 
sponding to the positions of the lenses, before the data 
is multiplexed to place under the microlenses. Alterna- 
tively, the lattice of microlens positions may be broken 
down into several subsets of points on a square lattice, 
and several sets of orthographic images may be ren- 
dered, to correspond to the lens positions in each lattice. 
[0036] Filtering and resampling of the image data be- 
fore or after multiplexing may also be used to apply 'anti- 
aliasing 1 . For example, the orthographic images may be 
rendered to a higher resolution than that corresponding 
to the number of microlenses in the image, and each 
orthographic image may be low-pass filtered and resa- 
mpled to the required resolution, before the pixels are 
multiplexed. More orthographic images than there are 
pixels under each lens may be rendered, and the imag- 
es under each microlens (after the pixels are multi- 



plexed), maybe low-pass filtered and resampled. Either 
or both of these approaches may be used to obtain high- 
er quality images. 

[0037] It has further been appreciated that there is 

5 normally a limit to the rate of change of brightness with 
viewing angle that a display such as this can provide in 
a satisfactory manner. Generally, the main limitation will 
come from the limited number of pixels under each lens, 
but other effects such as diffraction and spherical aber- 

10 ration will also contribute. If the display attempts to re- 
produce a high rate of brightness change by having 
large differences between adjacent pixels under a lens, 
then the effect will be to introduce a noticeable 'jump' in 
the observed image as the viewing position is moved. 

15 This can be thought of as an aliasing process, due to 
having too low a sample density. It is therefore desirable 
to generate images in which the rate of change of bright- 
ness is limited, to avoid noticeable jumps between im- 
ages corresponding to adjacent pixels; the feature of 

20 limiting the rate of change of brightness between pixels 
may be provided independently. This can advanta- 
geously be achieved by applying a spatial low-pass filter 
to each orthographically-rendered image, before the im- 
ages are multiplexed. However, the degree of filtering 

25 that is required is preferably dependant on the depth of 
the object in the image, in the same way that the depth- 
of-field of a normal camera introduces blurring that de- 
pends on the distance from the object to the camera. 
[0038] In order to address this problem, it is possible 

30 to adapt techniques used to simulate depth-of-field in 
the computer-rendering of normal perspective images; 
the simulation of depth-of-field in orthographic projec- 
tion may be provided independently. To simulate depth- 
of-field in perspective rendering, a known method is to 

35 render a number of views of the scene from different 
viewing positions across the aperture of the camera be- 
ing simulated. Each view is rendered with a slightly dif- 
ferent viewing angle, chosen so as to make the object 
that is in focus appear at the same place in each ren- 

40 dered image. The resulting images are added together 
to yield an image with simulated depth-of-field. To apply 
this method to the use of orthographic rendering for a 
3D display, a number of orthographic images can be 
rendered for each viewing angle, forviewing angles cen- 

45 tred on this angle. This process is shown for one viewing 
angle in Figure 5. If the horizontal viewing angles for 
adjacent pixels are A, A+d, A+2d, A+3d, ,...,thento gen- 
erate an "anti-aliased" image appropriate forviewing an- 
gle A, images should preferably be rendered at a 

50 number of horizontal angles ranging from A-d/2 to A+d/ 
2. The number of horizontal viewing angles for which 
images should be rendered and added together de- 
pends on a number of factors including the number of 
pixels under each lens and the range of depths in the 

55 scene. Preferably, at least 4-8 different angles should 
be used. If the display uses spherical microlenses (or 
some other method of presenting a different image as a 
function of vertical viewing angle), then equivalent anti- 
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aliasing processing should be applied vertically as well. 
For example, 64 different images might be rendered for 
each view, with 8 horizontal angles and 8 vertical angles. 
From Figure 5 it will be apparent that objects positioned 
closetothe plane of the display will appear in essentially 
the same place in each image, and therefore will not be 
blurred when the images are added together. Converse- 
ly, an object at the far end of the viewing volume will 
appear in a slightly different place in each rendered im- 
age, and therefore be blurred when the images are 
summed. 

[0039] It will be appreciated that "depth of field" is not 
a criterion that naturally applies to orthographic projec- 
tions; an orthographic projection can be considered to 
be a view from infinity with all objects at the same depth. 
Nevertheless, simulating depth of field for an ortho- 
graphic projection is useful in the present application. 
This may be provided independently. 
[0040] Other methods may also be used to reduce the 
resolution of objects at a large distancefrom the display. 
For example, the viewing volume of the orthographic 
projection could be divided up into a number of zones, 
by positioning the near and far clipping planes at differ- 
ent locationsthroughthe volume. Each zonecan be ren- 
dered separately, at a different resolution, such that 
zones containing objects that are furtherfrom the plane 
of the display are rendered at a lower resolution. The 
images rendered for each zone can be combined, taking 
care to ensure that objects nearer the front of the view- 
ing zone obscure those in zones further back. This can 
conveniently be achieved using OpenGL by rendering 
the furthest zone first, and then rendering subsequent 
zones into the same frame buffer without first clearing 
the buffer. The normal process by which values in the 
Z-buffer are checked to control whether a newly-ren- 
dered pixel should be allowed to overwrite one rendered 
previously will ensure the correct depth priority. 
[0041] Similar filtering and resampling techniques 
may be applied to allow the use of a smaller number of 
orthographically-rendered images than there are pixels 
under each lens, by interpolating additional pixel values 
in the images under each microlens. Equally, the reso- 
lution of each rendered orthographic image may be low- 
erthan the number of microlenses in the image, and ad- 
ditional pixels interpolated in each orthographic image 
to match the number of microlenses. 
[0042] The described techniques of filtering and resa- 
mpling the othographically-rendered images can be ex- 
tended to the representations provided by the known 
"real" integral image camera. Thus, such a representa- 
tion formed with an N x M array of microlenses can be 
de-multiplexed to provide a set of orthographic projec- 
tions. The number of such projections will be determined 
by the number of pixels under each microlens. The res- 
olution of each projection will be N x M pixels. Each of 
these demultiplexed projections can then be subjected 
to a variety of conventional video processing tech- 
niques, as if they were true images. It would be possible, 



for example, by resampling to provide a set of images 
having different resolution, say P x Q. Remultiplexing 
would then provide a 2D representation which can be 
viewed under a P x Q microlens array. In other senses, 

5 the availability of demultiplexed othographic projections 
will enable types of processing which would otherwise 
be impossible or impracticable, or will improve signifi- 
cantly the performance of aspecific technique. Thus, an 
orthographic projection will be expected to share the 

10 spatial correlation of a real image. Compression tech- 
niques such as JPEG and the intra-coding within MPEG 
should therefore be high ly efficient. A compression tech- 
nique which first demultiplexed the representation into 
a set of orthographic projections, should therefore be 

15 more efficient than one operating on spatial blocks of 
the representation itself. Related advantages will be 
found with other spatial processing techniques such as 
noise reduction. Similar effects should be seen in the 
temporal correlation of successive pictures. Thus, a 

20 succession of orthographic projections can be proc- 
essed as if it were a video signal. 
[0043] Although the invention has been described in 
terms of its application to a display using microlenses, 
it can be applied to any type of 3D display where each 

25 small area of the screen appears to have a brightness 
and colour that varies as a function of viewing angle. 
Such displays include those using multiple displays or 
high frame-rate displays with an associated mechanism 
to project particular images at particular angles. 

30 [0044] A particular application is in the formation of 
moving 3D images. In such a case, the set of images is 
generated to form a sequence of images. This can be 
achieved using a conventional high speed graphics 
workstation. If the number of viewing angles is relatively 

35 small and the display size is not too large, the images 
may be rendered in real time to produce an acceptable 
frame rate; otherwise the images may be pre-rendered 
and stored. The workstation may include a conventional 
communication interface for receiving data over a net- 

40 work, such as the internet, or a receiver for receiving 
data over a broadcast link, to produce images for dis- 
play. Since the hardware may be largely conventional, 
other than the modifications to the display to effect de- 
multiplexing (e.g. an array of microlenses to effect spa- 

45 tial demultiplexing) it is not depicted. 

[0045] Whilst the invention finds use in moving imag- 
es, it is also useful in generating static images, for ex- 
ample advertising, posters, promotional objects and the 
like. For a static image, processing constraints are less 

50 severe and higher resolution output may be more cost- 
effectively available, for example using colour laser or 
ink jet printers and so the advance in processing speed 
may be used to produce physically large images, for ex- 
ample for an advertising hoarding, having a large range 

55 of viewing angles. In static images, the microlens array 
may be laminated onto the printed image representa- 
tion. 

[0046] Each feature disclosed herein may be inde- 
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pendently provided, unless otherwise stated. 
Claims 

1. A method of providing a representation of a 3D 
model, comprising the steps of identifying a set of 
different viewing angles and rendering from said 
model a set of orthographic projections correspond- 
ing respectively with said viewing angles. 

2. A method according to Claim 1 , wherein the set of 
orthograpic projections is spatially multiplexed to 
provide a set of micro images, each micro image 
containing one pixel from each of the projections. 

3. A method according to Claim 2, further comprising 
registering the micro images with respective micro- 
lenses of a lens array to demultiplex the images. 

4. A method according to Claim 3, wherein the repre- 
sentation is a hard copy representation. 

5. A method according to Claim 2 wherein the images 
are rendered on a 2D display registered with an ar- 
ray of microlenses. 

6. A method according to Claim 5 wherein the 2D dis- 
play is selected from the group consisting of a cath- 
ode-ray-tube, an LCD panel or a plasma panel dis- 
play. 

7. A method according to Claim 5 or 6, wherein the 
display is pre-registered with the array of microlens- 
es. 

8. A method according to any preceding claim, further 
comprising generating a subsequent representa- 
tion to produce a sequence of representations. 

9. A method according to Claim 8 wherein the repre- 
sentations are generated at a display site from a se- 
quence of 3D models transmitted from a source 
over a communication or broadcast link. 

10. A method according to Claim 8 wherein at least a 
portion of the representation is pre-generated at a 
source and transmitted over a communication or 
broadcast link to a display site. 

1 1 . A method according to any preceding claim wherein 
the number of rendering operations corresponds to 
the number of viewing angles. 

12. A method according to Claim 11, wherein the 
number of rendering operations is less than the 
number of pixels in an individual image. 



1 3. A method according to any preceding claim wherein 
the 3D object includes elements on both sides of 
the image plane. 

5 14. A method according to any preceding claim, further 
comprising presenting the images for view so that 
an image appearing to have perspective qualities is 
formed from portions of a plurality of different ones 
of said orthographic projections. 

10 

15. A method according to any preceding claim further 
comprising filtering the images to reduce sudden 
variation in brightness as viewing angle changes. 

15 16. A representation of a 3D scene or model for display 
utilising 2D display means, comprising a set of 2D 
images representing respective orthographic pro- 
jections of the 3D scene or model at respective dif- 
ferent angles. 

20 

17. A method of processing a viewable representation 
of a 3D model or scene, comprising the steps of de- 
multiplexing the representation to form a set of or- 
thographic projections corresponding respectively 

25 with different viewing angles, and processing each 
projection. 

18. A method according to Claim 1 7, wherein the proc- 
essed projections are remultiplexed. 

30 

19. A display device for providing a representation of a 
3D model, comprising a processor for rendering 
from said model a set of orthographic projections 
corresponding respectively with a set of different 

35 viewing angles and a display element for producing 
a visible display of the representation. 

20. A display device according to Claim 1 9 wherein the 
display element is substantially flat. 

40 

21 . A display device according to Claim 20, wherein the 
set of orthograpic projections are spatially multi- 
plexed to provide a set of micro images displayed 
on the display element, and wherein a plurality of 

45 microlenses are registered with the micro images. 

22. A display device according to Claim 21, wherein 
each micro image comprises a single pixel from 
each orthographic projection. 

50 

23. A display device according to any of Claims 19 to 
22 further comprising means for receiving the 3D 
model or a representation over a communication 
link. 

55 

24. A computer program or computer program product 
containing instructions for carrying out a method ac- 
cording to any of Claims 1 to 1 5. 
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Figure 1 
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lenticular lens array over multiple cameras and 
display corresponding images 



Figure 2 
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Microlenses and image 
obtained by multiplexing 
orthographic camera images 



Figure 3 



11 



EP1 143 747 A2 




lenticular lens array over 
display 



Figure 4 
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volume imaged by an model 
orthographic camera without 
depth-of-field processing 
(shaded) 
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